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ABSTRACT . / 

Concern for the consequences of educational programs 

as well as their processes is the focus of this study. The point is 
made that most evaluations of basic education programs are inadequate 
for two reasons: (1) The economic aspects of remedial education for 

the disadvantaged are infrequently studied; and (2) Most evaluations 
are based on unsophisticated research designs. It is also pointed out 
that to society, the primary economic benefit of basic education 
programs is the increase in trainee earnings during the posttraining 
period, which are directly attributable to remedial educational 
instruction. Three problems, however, which must be considered in 
measuring these benefits are: (1) Distinguishing between permanent 

and transient changes in earnings, (2) Married women with no need or 
desire to work, and (3) A worker who chooses to reduce his work 
effort. Possible evaluation designs include: the income equivalency 
approach, the before-and-after design, the quasi-experimental 
approach, and the true experiment. It is concluded that basic 
education programs should be designed to include an evaluation 
component as part of their normal administrative procedure. It is 
also concluded that the knowledge gained from evaluation should be 
f unneled back into the program to produce more effective program 
operation. (Author /CK) 



^3 ED 062 601 



«j v " 






U.S. OEPARTMENT OF HEALTH. 

EOUCATION & WELFARE 
OFFICE OF EDUCATION 
THIS DOCUMENT HAS BEEN REPRO- 
DUCED EXACTLY AS RECEIVEO FROM 
THE PERSON OR ORGANIZATION ORIG- 
INATING IT. POINTS OF VIEW OR OPIN- 
'IONS STATEO 00 NOT NECESSARILY 
REPRESENT OFFICIAL OFFICE OF EOU- 
CATION POSITION OR POLICY. rrrrt 

EDUCATION PROGRAMS FOR ADULTS: 
CONCEPTUAL AND METHODOLOGICAL PROBLEMS* 



EVALUATING BASIC 



SOME 



by 

■X'X 

Myron Roomkin 



Adult educators have always been concerned with the effectiveness of 
basic education programs for. adults. Even a cursory reading of the litera- 
ture reveals an enormous amount of research dealing with such aspects of 
program operation as pupil recruitment, instructional techniques and curric- 
ulum design. Recently, however, there are indications that the focus of 
research in this field is shifting. Although we are still concerned with 
developing effective methods of instruction, we are also beginning to exam- 
ine the consequences of remedial education programs. That is, we are con- 
cerned with the consequences of programs as well as their processes. This 
newer research focus is often called program evaluation research. 

As an adult educator, I welcome the new concern for program evalua- 
tion. But as an economist, I find most evaluations of basic education 
programs to be inadequate, for two reasons. First, the economic aspects of 
remedial education for the disadvantaged are infrequently studied; and 
second, most evaluations are based on unsophisticated research designs. 
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Under the assumption that adult educators will face continued en- 
couragement from funding agencies to conduct evaluations, this paper dis- 
cusses these issues and offers some recommendations for upgrading our eval- 
uative efforts . The premise of the paper is that evaluators of basic educa' 
tion can learn a great deal from previous evaluations of government-funded 

vocational training programs, conducted under the Manpower Development and 

2 

Training Act and other legislative authorizations. 



Economic Aspects of Basic Education 
It is doubtless true that basic education programs have as their 

immediate goal the improvement of individual levels of educational achieve- 

„ 3 

ment. However, most of the 28 different, federally- funded programs are 

based on the proposition that improved achievement will translate directly 

into economic gain through the mechanism known as the labor market. In 

this sense, basic education programs are, to varying degrees, training pro 

grams in the broadest of occupational skills — general academic education. 

Because of its relationship to employment and earnings, basic education 

shares similar goals with other government-funded manpower development and 



policies and programs. 
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A compilation of these studies can be found in Gerald G. Somers, 
Retraining the Unemployed (Madison, Wisconsin: University of Wisconsin Press, 

1968) . A summary of the principal evaluation studies of training programs 
is presented by Einar Hardin, "Benefit-Cost Analyses of Occupational Training 
Programs: A Comparison of Recent Studies," in G. G. Somers and W. D. Wood 

(eds.), Cost Benefit Analysis of Manpower Policies. Proceedings of a North 
American Conference (Kingston, Ontario: Hanson and Edgar, Ltd ., l$k>9 ) , 

pp. 97-118. 



^For a complete listing of federally- funded basic education programs, 
see Greenleigh Associates, Inc., Inventory of Federally Funded Adult Basic 
Education Pro, ‘trams: Report to the President's National Advisory Comittee 

(Now York: G re onlo igh As noc 1 at o Inc., lyoo) . 
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As I have argued elsewhere: 

Given the Federal, government's involvement in basic education 
programs — in initiating, financing, and monitoring programs — 
it is appropriate for the government to determine the economic 
efficiency of these programs. It is an economic fact of life 
that federal funds for anti-poverty programs [and I would add 
manpower programs] are limited, while at the same time, programs 
such as institutional occupational training, on-the-job instruc- 
tion and even income maintenance schemes are offered as alter- 
natives to basic educational instruction.^ 

i 

Thus, the burden falls on adult educators to demonstrate a satisfactory level 
of economic performance at minimal costs, compared to alternative programs 
and policies. In the jargon of economics, the benefits and costs of basic 
education must be identified and measured in similar units, usually constant 
dollars . 

As a word of caution, intuitive definitions of program benefits and 
costs may be erroneous. Previous evaluations of government- funded training 
programs have found it useful to specify different evaluative contexts when 
distinguishing between these concepts. Much depends on whether an evaluator 
approaches the problem from the perspective of society as a whole, the in- 
terests of an administering agency, or the viewpoint of individual program 
participants. 

Programs financed with public funds must be evaluated in terms of 
their contribution to general welfare and the costs of alternative uses of 
public funds. Similarly, a rational individual must choose between many al- 
ternative paths to self-improvement. These, then, are private gains which 
should be compared with private costs, that is, the cost to individuals for 



Myron Roomkin, "The Benefits and Costs of Basic Education for Adults: 
A Case Study," jn U. S. Congress, Joint Economic Committee, (tentatively 
titled) A Compendium of Papers on Cost-Benefit Analysis , forthcoming. 
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participating in basic education instruction. The economic considerations 
of an administering agency are often different than those of either society 
as a whole or individual trainees. The agency context, however, will also 
vary between programs, making generalizations almost impossible. Conse- 
quently, the following discussion is limited to identifying social and pri- 

* 

vate benefits and costs. 



Social Economic Benefits and Costs 

To society, the primary economic benefit of basic education programs 
is the increase in trainee earnings during the posttraining period, which 
are directly attributable to remedial educational instruction. Such a gain 
could be the result of increased hourly wage rates or increased hours of 
employment. Society would also view as beneficial any increase in post- 
training employment, or conversely, any reduction in the average unemploy- 
-raent rate for members of the target population. Moreover, an acknowledged 
economic benefit of government training programs of all types (but one which 
is not easily measured), is the impact of training on the stability of 
prices, i.e., the rate of inflation. If inflation is partly the result of 
labor market shortages, then educational programs can have some effect on 
increasing the supply of labor at given quality levels. A crude proxy for 
this labor supply effect is the increase in labor force participation rates 
displayed by trainees. Finally, because remedial instruction can lead to 
continued training in occupational skills, the extent to which graduates of 
basic education programs enter skill training programs is also a direct 



benefit for society. 

/ 

» 

However, the measurement of these benefits often involves confronting 
difficult conceptual problems. I have selected three such problems which are 
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particularly thorny. First, in assessine the level of trainee earnings in 
the .posttraining period, we must distinguish between permanent changes in 
earnings and those which are of a transient variety. Only changes in earn- 
ings that persist for reasonable periods of time and that withstand changing 

labor market conditions should be counted. 

Second, females pose special problems in the measurement of program 

benefits. A married woman with no desire or need to work will display zero 
scores on all economic benefit measures. Aggregate analysis of program 
participants regardless of sex may understate the economic benefits of those 
workers fully committed to the labor force. This problem requires that ben- 
efits be computed on an average basis for each sex group with further dis- 
tinctions made for noncommitted, partially committed and fully committed 

workers. 

Finally, a rational worker, who receives additional earnings as a 
result of remedial education, may choose to reduce his work effort, that is, 
maintain his total weekly earnings at a previous level but increase his lei- 
sure time. Thus, it is important to distinguish between an hourly wage ben- 

efit and a total earnings benefit. 

As for the social costs of programs, we can identify the following 

cost items which should be included in a calculation of total social cost: 

1 . Total operating or instructional costs; 

2 . Total cost of allowances paid trainees while in training; 

3. Total opportunity costs or foregone earnings of trainees while 
in training, since their earnings are lost to society, 

4 . Prorated expenses of shared, facilities such as buildings, 

5. The cost of idle capacity resulting from loss than 100 per 
cent, utilisation of instructional resources. 
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Some have argued that the opportunity costs of training programs are 
zero because any job vacated by a trainee at the time of his entrance would 

i 

be filled by someone of comparable quality out of the ranks of the unemployed. 
But in fact, this is an empirical question. Not all trainees have prior em- 
ployment, and many employers often do not fill vacancies. 

Private Benefits and Costs 

The costs and benefits for individual participants in basic education 
programs require somewhat different calculations. Private benefits usually 
include the after-tax income of trainees, minus any redaction in transfer 
payments (i.e., government assistance payments such as unemployment, com- 
pensation and welfare payments) . 

The private costs of instruction consist only of those costs of in- 
struction borne directly by the trainee. These should include the foregone 
earnings of a trainee while in training, minus his living or trainee allow- 
ance plus the costs of training-related expenses (i.e., transportation ex- 
penses or school supplies) . Since stipends or trainee allowances are often 
sizable--and frequently larger than the foregone earnings of trainees — we 
can see how private benefits often compare more favorably to private costs 
than social benefits do to social costs. In a case study evaluation of 
basic education financed under the Manpower Development and Training Act, 

the size of the stipend was found to be an important .reason for trainee 

S 

participation in the program. 



Tbid. 



o 

ERIC 



5 



p 



j J 

7 j 

i 

! 

Relating Benefits to Costs . j 

Both from the social or private viewpoint, knowledge of either the ! 

i 

! 

benefits or the costs is inadequate for evaluation pruposes. Economic ev&l- | 

uations require that costs and benefits be compared within an appropriate ' 

framework. 

The first step is to project both costs and. benefits over an appro- 
priate time horizon. For example, an increase in the posttraining hourly 
earnings of trainees, directly attributable to basic education should be 

summed over the many future hours of the individual’s work life. This pro- 

6 

cedure is known as taking the present value of the benefit stream. Typical- 
ly, only the benefits stream requires this calculation, since al j costs of 
basic educatioji are usually borne during the life of the training program. 

Armed with the present value of benefits and costs, a comparison 
between the two can be made in many different manners. First, total or aver- 

/ 

age benefits can be divided by total or average costs respectively, producing , 

a benefit-cost ratio. Ratio values less than one are considered very poor 
performance from both the social and private perspectives. Ratios equal to 

t 

one should not be looked on with pride either, for they indicate that, other 
things being equal, society or trainees could have benefited equally from a 
direct cash subsidy in lieu of education. Rather, ratios exceeding one in- 
dicate good program performance; the larger the value the better. 
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A formula for computing the present value of a benefit or a cost stream 
is: 



where 
and i 



N 

PV a z Y .0- + i)“ J 
j : -0 J 

j is the lengt h of the stream, Y . is 

J 

is an appropriate rate of discount. 



the yearly benefit 



or cost 
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More sophisticated methods for canparing costs and benefits require 
the computation of a program rate of return. As businessmen compute a re- 
turn on their investment in physical capital, the government or an individual. 

7 

can compute a return on an investment in human capital. Alternatively, 

we can assume a value for the rate of return and calculate the time required 

8 

to recoup this investment. 

Quite frequently the goals of basic education programs cannot be ex- 
pressed in dollar equivalents, but it is still possible to relate such as- 
pects of program performance to cost considerations. As long as we have 
confidence in our measures and can develop ratio scales, the effectiveness 
of a program can be assessed within a cost-effectiveness framework. Any 
index of program performance, "P," can be related to program costs, "C," 
in ratio form, such that 

AP 

*£- = a cost per imit of change in P . 

Interestingly, "P" may also be a probability, measuring, for example, the 
conditional likelihood that a graduate of basic education will continue his 
education during the posttraining period. 



The rate of return can be found by solving for r in: 

n i 

0 = Z (Y 1 - X.)(l I r)~ d 

. j=0 J J 

where Y. = the benefit in year j, X . = the cost in years, j = the length 
J J 

of the stream and r = the rate of return. 

8 

Solving for j in the above equation, we can compute a "payback 
period" to the investment. 
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The Methodology of Evaluation 



• The literature dealing with program evaluations in all disciplines is 

voluminous. This discussion, therefore, highlights only a few Important is- 

9 

sues. 



Estimating Benefits 

To produce accurate estimates of program benefits an evaluative ef- 
fort should (1) select an appropriate research design, (2) specify and col- 
lect appropriate data, (3) isolate basic education's effects, and (4) re- 
ject alternative causal factors. Previous evaluations of basic education 
programs, even those not dealing with essentially economic considerations, 
often fall short on one or all of these requirements. 

Design. Evaluators have used four different research designs to eva- 
luate basic education. The first, and perhaps simplest design can be called 
the income equivalency approach. Examples of this design can be found in 
the work of Roomkin 10 and Levin and Slavet. 11 The income equivalency ap- 
proach assumes that grade levels of educational achievement, measured on 
standardized achievement tests, are reasonably equivalent to grade levels 
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tion, University of Wisconsin, 1971) . ’ Wlsc0nsln (Ph.D. disserta- 
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Melvin R. Levin and Joseph S. Slavet 
Ky.: Health Lexington Books, lqyo), pp . ’ 
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of foraial educational attainment. Thus, trainee achievement gain can be 
translated into annual earnings gain by computing the dollar equivalency 
in annual earnings associated with a specific change in educational at- 
tainment for a known population. 

The inadequacies of this design are obvious. Achievement is not nec 
cessarily a perfect proxy for attainment. Second, educational achievement 
scores are often based on unvalidated or -oorly validated tests. Third, de- 
tailed data on earnings and education are not always available, and avail- 
able data are not always current. But the design's biggest weakness is its 
reliance on an untested proposition: Trainees will succeed in the labor 

market in direct relation to their within program performance. Findings 
stemming from this approach beg further evaluative efforts. The approach 

should only be used where evaluators must function under the most stringent 
research constraints. 

A second, frequently used approach is the before-and-after design. 
Trainees are studied after a reasonable time period and their poettraining 
status is compared to their pretraining status. Patten and Clark , 12 Pearce 13 
and Kohler and Seaman* have conducted case study evaluations using the 
before and-afte r comparison. Though more rigorous than the income 

12 

Thomas Patton. Jr# and p pi ^ > T . 

Job Placement of Hard-Core Unemployed in Detroit ^’r Llt ® rac ^ lining and 
Vol. 3, No. 1 (1968) f pp, 25~1»6^ nal of HujTian Resources . 

13 t 

June, 1966Simcogr^hed.) Wt Education: Evaluation Through Research,” 

Description of Adult Ba s 5 (TlLuc a t io n ^ tu 1 "nl. ^ Ed V ca ^ ion;a a | ld Socioeconomic 
Human Resources ^ 511-5^. “ Journal of 
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'equivalency approach, the before-and-after approach fails .to answer one key 



the program? 

The third design, the quasi- experimental approach, though not a per- 
feet procedure, allows us to at least address this question. As utilized 



the System Development Corporation, the quasi -experimental design employs 
a specific group of nontrainees, sometimes called a control or comparison 
group, to determine the level of trainee performance in the absence of remedial edu- 
cation. What fault there is with this approach stems from our inability to 
produce a control group (perfectly) matched to the trainee group on all rel- 
evant characteristics and variables. 

Under the fourth design, the true experiment, the control group con- 
tains individuals who are perfectly matched to the experimental, or treat- 
* ment group. I have identified only one instance where a true experimental 

iO 

evaluation of basic education was conducted. In a study by Brazziel, en- 
rollees were randomly assigned to groups. Interestingly, Brazziel 1 s study 



15 

Roomkin, "The Benefits and Costs of Basic Education for Adults: A 

Case Study, " op. cit . 

^Greenleigh Associates, Inc., Field Test and Evaluation of Selected 
Adult Education Systems (New York: Greenleigh Associates, Inc., 1966) . 

17 

‘System Development Corporation. "A Longitudinal Evaluation of Adult 
Basic Education Programs " (in progress) . 

18 

William Brazziel, "Effects of General Education in Manpower Programs," 
Journal of Human Resources , Vol. 1, No. 1 (19 66), pp. 39 -M. 



question: What would have happened to trainees had they not participated in 
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employed a placebo group-u sort of no-truining, training program. De- 
sirable as is the true experimental approach, a widespread adaptation of 
this design is very unlikely. More often than not, evaluators Action 
after the fact, usually after trainees have left the program. Furthermore, 
the assignment of individuals to control or treatment groups on a random ’ 
basis raises serious questions of equity in program administration. 

In the absence of random assignments, the best control group is one 

containing nontrainees with the identical characteristics and qualities of 

trainees, selected from a population of individuals, who had equal knowledge 

of, and access to, remedial educational programs. A control group should 

be matched to the treatment group on the following characteristics: 

1. age 



2. sex 



3* race 



4. employment status before training 
educational attainment 

6. educational achievement (if possible) 

7. motivation or desire to succeed. 

The last two of those, of course, will be difficult to measure-but, I stress 

their importance on the basis of experience. In the study of trainees, under 

the Manpower Development and Training Act previously noted, 1 ? these variables 

were not used for matching purposes and motivational difference between the 
groups biased the results. 



19 



Myron lioomkin, "An evaluation of Adult 1 u 



power Development and 'Training Act "in 1: ' ducut ;^ on imder the Man- 

Myron Koomkin, "Kmployment and Kariiln*-. biTect- S!°" S1 |u ^ cL .£- Also sc 
I'br Milwaukee MDTA Kxpericnce (Mimeo-raphed ) ' ^lueution for Adults: 



eo 



ii 



13 



Frequently evaluators select program dropouts or no-shows to serve 

as a control group. It should be remembered, however, that dropping out of 

a basic education program is not analogous to premature termination from a 

vocational training proeram. Basic education dropouts, by definition, have 

• • 

received some instruction and are therefore a poor basis for comparison. 
Second, as learned from the evaluations of skill training programs, many 
dropouts leave to accept employment. And some leave because they lack ade- 
quate motivation. In the final analysis we have very little control over the 
quality or quantity of dropouts or no-show trainees . 

Alternative sources of control groups should be developed. In fact, 
wherever possible, multiple control groups should be employed. Some suc- 
cess has been achieved with control groups randomly selected from the files 

. * 20 

of the local Employment Service or welfare department. In a quasi- 
experimental design, no control group is perfect, jlowever, if we under- 
stand the nature of the bias, we can control for it. 

Data Specification and Collection . Evaluation starts with baseline 
data, generated by the program administrators themselves. Experience sug- 
gests that enrollment and performance records of students are not always 
verified and up-to-date. There is no substitute for valid information in 
the evaluation process. 



20 See Glen C. Cain and Gerald G. Somers, "Retraining the Disadvan- 
taged," in Catklccn Quirk and Carol Sheehan (eds.), Research in _ Vocation al 
and Technical Education (Madison, Wisconsin: Center for Studies in Voca- 

tional and Technical Education, University of Wisconsin, 19°7), PP- 
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The list of important information needed for evaluation is endless, 
but the resources and opportunities for measurement are limited. A prudent 
determination of important variables is required. Of highest priority are 
measures of performance, including economic performance indicators. Some 
efficient allocation of research resources can be achieved by selecting 
variables for study on the basis of their theoretical importance. 

For postprogram information, most evaluations employ the follow-up 
method of data retrieval and use personal interviews. Though costly, this 
seems to be the only viable way to obtain longitudinal data on both control 
and treatment samples. In specifying the length of . the follow-up period, 
researchers must balance three concerns: 

1. a desire to measure permanent rather than transient performance; 

2. a preference for high rather than low response rates; and 

3. a recognition that program graduates enter continuing educa- 
tion programs . 

I would also urge that former program participants be retested to de- 
termine' the degree of educational achievement retention. 

Finally, because we must use survey research techniques, some reduc- 
tion in sample size will be encountered during the follow-up period. The 

1 

acts of locating and interviewing persons in this target population are es- 
pecially problematic. 

Isolating the Effects of Basic Education . Rigorous analysis of col- 
lected data requires using multivariate statistical techniques. Multiple 
regression analysis, because it yields estimates of the sign and magnitude 
of relationship, is, in my opinion, the most desirable technique. Other 
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statistical techniques such as discriminant analysis, analysis of variance 
and factor analysis should be used where appropriate. 

Within a regression framework, the simplest evaluation model for 

* 

basic education is: 




N 

a + bY + E c.X. + dE + p 

t • -i i i 
i=l 



where Y and Y are measures of performance indicator Y in pre- and 
t t+1 

posttraining periods, t and t+1, respectively; a is a constant, EX^ 

i 

is a set of control variables, usually including age, sex, race, educational 
attainment and time exposed to the labor market; E is a dummy variable 
representing program participation, and p is the error term. The regres- 
sion coefficient on E, d, is our best guess estimate of the benefit re- 
sulting from basic educational instruction. 

The model, however, should be elaborated to reflect other relevant 
conditions. For example, a large number of basic education trainees continue 
their education after completing remedial educational instruction. A re- 
gression model reflecting such behavior takes the following equational form: 

N 

y = a + bY + E c.X. + dE + eT + f(E X T) + p . 
t+1 t i=1 i i 



By adding variables T, representing vocational training, and T x E, 
representing combined basic education and vocational training, we are able 
to isolate the effects of these additional training programs on the depen- 
dent variable. 
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Multiple* regression analysis also offers a direct test of control 
group-treatment group comparability. As indicated even a control generated 
through random assignments could lose its appropriateness during the survey 
portion of a study. Through regression analysis, a test for group compara- 
bility can be performed, by estimating three separate regression equations: 

N 

+ bY + 2 c.X. + n 
t i=l 1 1 

N 

+ bY + 2 c.X. + lx 
Z i=l 1 1 

N 



(1) 


1! 

cT + 1 


(2) 


K 

c+ 

,+ w 
11 


( 3 ) 


(C+BE) 

t+1 


where superscripts 


C, BE, 



i=l 



l l 



M* 



bined samples of both samples, respectively. The hypothesis is that the cor- 
responding coefficients in each equation— a, b, and c— are equal. Or, 

phrased verbally, both control and treatment samples are drawn from the same 

PI 

larger population. 

Al ternative. Hypotheses . Like any statistical tool, regression pro- 
cedures are as valid as the underlying assumptions and model on which they 
are based. The exclusion of a key variable in the above equations will 
drastically bias the results. It is for this reason competing explanations 

of the differential between program participants and control observations 
should be explored. 



21 * 

r 4 General description of the test can be found in J. Johnston 

Econometric Methods (New York: McCraw-Hi ll 1963) p 136 ’ 
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The literature on evaluation suggests the following competing hy- 
potheses. 

1. Trainees gain economically because they displace nontrainees 

in the labor market. Thus, what is gained by one group is 
% 

lost by another. This is called the displacement effect. 

2. The status of nontrai'nees was improved when trainees left the 
labor market. This is called the vacuum effect. 

3. Trainees gain because they have special access to information 
on training programs and job opportunities. The importance of 
this factor should be closely examined when programs are coor- 
dinated by local offices of the Employment Service. 

4. Trainees gain from the certification of a program rather than 
from any true quality changes. We are beginning to call this 
credentialism or a sheepskin effect. 

Measuring Costs 

Typically costs are easier to measure. Most are available directly 
from budget statements. However, opportunity costs will not appear on any 
budget sheet. These can only be secured directly from the control group. 

A faulty control group, therefore, will not only yield inaccurate benefit 
levels but poor cost estimates as well. 
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Some Problems in Benefit-Cost Analysis 

The calculation of present values and cost-benefit relationships re- 
quires many operating assumptions. For instance, the selection of an appro- 
ximate rate of discount or the projection of earnings streams into the future 
arc; oftentimes arbitrary decisions. It is for this reason that a set of 
benci it.-- cost calculations rather than a single calculation should be made, 

16 
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employing a wide range of varying assumptions . Multiple measures of pro- 
gram performance will give policy-makers flexibility in comparing the per- 
formance of many programs . 

Concluding Remarks 

The previous discussion has been heavily weighted towards economic 

considerations. Obviously, economics constitutes only one facet of basic 

» 

education for adults. However, the desire to utilize remedial education 
to obtain many different changes in the behavior of the disadvantaged, 
poses important problems to evaluators. Administrators must decide on an 
a priori basis the relative weight to be given each desired outcome. Thus^ 
poor performance in one area can be offset by adequate performance in others. 
In the absence of legislative directions, program administrators should 
reach agreement on what they consider to be the relative importance of pro- 
gram goals. 

In conclusion, whether or not my suggestions for the content and for- 
mat of evaluations are accepted, I urge that basic education programs be de- 
signed to include an evaluation component as part of their normal adminis- 
trative procedure. Program evaluation should be planned, just as we 
plan program curricula. Ample resources should be set aside to conduct an 
evaluation. Control groups should be identified early. Enrollment infor- 
mation should be kept current and all data should be verified. Finally, 

' evaluation is of little value if results are not utilized. At a minimum, 
the knowledge gained from evaluation should be funneled back into the program 
to produce more effective program operation. 
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